<html>
<head><meta charset="utf-8"><title>using snappy for metadata · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html">using snappy for metadata</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="207594259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207594259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207594259">(Aug 21 2020 at 01:58)</a>:</h4>
<p>Just got performance numbers back on <a href="https://github.com/rust-lang/rust/pull/75754">https://github.com/rust-lang/rust/pull/75754</a> - looks like it's a performance win across the board. It adds one new dependency to rustc, but it's a pure-Rust dependency.</p>



<a name="207594278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207594278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Notification Bot <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207594278">(Aug 21 2020 at 01:59)</a>:</h4>
<p>This topic was moved here from <a class="stream-topic" data-stream-id="131828" href="/#narrow/stream/131828-t-compiler/topic/using.20snappy.20for.20metadata">#t-compiler &gt; using snappy for metadata</a> by <span class="user-mention silent" data-user-id="239881">Josh Triplett</span></p>



<a name="207595881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207595881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207595881">(Aug 21 2020 at 02:43)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> not sure how stable this is, but: <a href="https://perf.rust-lang.org/compare.html?start=e15510ca33ea15c893b78710101c962b11459963&amp;end=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;stat=cycles%3Au">https://perf.rust-lang.org/compare.html?start=e15510ca33ea15c893b78710101c962b11459963&amp;end=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;stat=cycles%3Au</a></p>



<a name="207595938"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207595938" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207595938">(Aug 21 2020 at 02:44)</a>:</h4>
<p>it's too close to neutral on instructions in many cases for me to consider it "across the board", hence looking at other measurements. I wish we had a good way to tell when they're consistent enough to be telling and automatically warn us about them</p>



<a name="207596009"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596009" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596009">(Aug 21 2020 at 02:46)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> If I were to take a wild guess, I would <em>guess</em> that's because snappy is much faster but then generates more data that has to be written out, and it takes extra instructions to write the data out, but those instructions don't actually add to wall-clock time.</p>



<a name="207596020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596020">(Aug 21 2020 at 02:46)</a>:</h4>
<p>That's absolutely a guess, though.</p>



<a name="207596031"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596031" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596031">(Aug 21 2020 at 02:47)</a>:</h4>
<p>it's the other way around though? you're neutral/winning on instructions but increase on cycles, which means some of the instructions now take more cycles, i.e. are slower</p>



<a name="207596035"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596035" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596035">(Aug 21 2020 at 02:47)</a>:</h4>
<p>sadly none of the time metrics look obviously useful</p>



<a name="207596036"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596036" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596036">(Aug 21 2020 at 02:47)</a>:</h4>
<p>Ah, sorry, I thought what you were linking to was instructions.</p>



<a name="207596038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596038">(Aug 21 2020 at 02:47)</a>:</h4>
<p>Also, something else worth noticing:</p>



<a name="207596039"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596039" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596039">(Aug 21 2020 at 02:47)</a>:</h4>
<p>no, instructions:u is the default</p>



<a name="207596044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596044">(Aug 21 2020 at 02:47)</a>:</h4>
<p>I didn't realize that had become the default.</p>



<a name="207596083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596083">(Aug 21 2020 at 02:48)</a>:</h4>
<p>Every one of the cases that shows a regression is a -doc case.</p>



<a name="207596084"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596084" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596084">(Aug 21 2020 at 02:48)</a>:</h4>
<p>it's the only thing we look at most of the time. it's been the default for years I thought</p>



<a name="207596089"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596089" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596089">(Aug 21 2020 at 02:48)</a>:</h4>
<p>oh we can ignore <code>-doc</code>, but that's not my worry. let me look again</p>



<a name="207596092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596092">(Aug 21 2020 at 02:48)</a>:</h4>
<p>okay I missed that, thank you</p>



<a name="207596094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596094">(Aug 21 2020 at 02:48)</a>:</h4>
<p>Why can we ignore -doc?</p>



<a name="207596098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596098">(Aug 21 2020 at 02:49)</a>:</h4>
<p><code>rustdoc</code> is not yet tuned to be "measurable"</p>



<a name="207596104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596104">(Aug 21 2020 at 02:49)</a>:</h4>
<p>Ah.</p>



<a name="207596109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596109">(Aug 21 2020 at 02:49)</a>:</h4>
<p>In that case, yeah: everything "measurable" gets faster. ;)</p>



<a name="207596110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596110">(Aug 21 2020 at 02:49)</a>:</h4>
<p>I forget how much variance it has but the results are all over the place IIRC. <span class="user-mention" data-user-id="232545">@Joshua Nelson</span> might have more recent updates</p>



<a name="207596154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596154">(Aug 21 2020 at 02:50)</a>:</h4>
<p>rustdoc shouldn't be counted until it stops loading all extern crates at startup</p>



<a name="207596155"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596155" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596155">(Aug 21 2020 at 02:50)</a>:</h4>
<p>we should've added a <code>-doc</code> toggle but I guess we haven't yet</p>



<a name="207596157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596157">(Aug 21 2020 at 02:50)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> lol</p>



<a name="207596160"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596160" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596160">(Aug 21 2020 at 02:50)</a>:</h4>
<p>If that's the case, would it make sense for performance runs to exclude the -doc test cases for now?</p>



<a name="207596164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596164">(Aug 21 2020 at 02:50)</a>:</h4>
<p>Nevermind, yeah, exactly. :)</p>



<a name="207596165"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596165" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596165">(Aug 21 2020 at 02:50)</a>:</h4>
<p>we want the data but not to show it by default, I think</p>



<a name="207596169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596169">(Aug 21 2020 at 02:50)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs#L423">https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs#L423</a></p>



<a name="207596171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596171">(Aug 21 2020 at 02:50)</a>:</h4>
<p>it's mostly useful if you change <code>rustdoc</code> itself, I guess</p>



<a name="207596172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596172">(Aug 21 2020 at 02:50)</a>:</h4>
<p>yeah</p>



<a name="207596174"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596174" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596174">(Aug 21 2020 at 02:51)</a>:</h4>
<p>it had -68% for the everybody_loops PR lol</p>



<a name="207596181"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596181" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596181">(Aug 21 2020 at 02:51)</a>:</h4>
<p>admittedly that was for the wf_traits stress test but still</p>



<a name="207596191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596191">(Aug 21 2020 at 02:51)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I'm actually surprised metadata compression isn't way worse, since we lose a crucial benefit: memory-mapping the <code>.rmeta</code> file inside an <code>.rlib</code> (assuming LLVM isn't pulling our leg here I guess) or a whole <code>.rmeta</code> file from an <code>--emit=metadata</code> build</p>



<a name="207596230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596230">(Aug 21 2020 at 02:52)</a>:</h4>
<p>errr</p>



<a name="207596235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596235">(Aug 21 2020 at 02:52)</a>:</h4>
<p>not sure I follow what that has to do with rustdoc</p>



<a name="207596236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596236">(Aug 21 2020 at 02:52)</a>:</h4>
<p>Zulip's tab complete is broken if you're not properly awake</p>



<a name="207596247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596247">(Aug 21 2020 at 02:53)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> oh you're changing the algorithm, not compressing metadata in real-world scenarios heh</p>



<a name="207596249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596249">(Aug 21 2020 at 02:53)</a>:</h4>
<p>We're already compressing metadata; this change just switches from deflate to snappy.</p>



<a name="207596251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596251">(Aug 21 2020 at 02:53)</a>:</h4>
<p>this should actually be entirely perf-neutral</p>



<a name="207596253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596253">(Aug 21 2020 at 02:53)</a>:</h4>
<p>since AFAIK we don't track <code>dylib</code>s on perf</p>



<a name="207596297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596297">(Aug 21 2020 at 02:54)</a>:</h4>
<p>well, "real-world modulo <code>rustc</code> itself which we don't measure"</p>



<a name="207596298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596298">(Aug 21 2020 at 02:54)</a>:</h4>
<p>Seems like we must be doing <em>something</em> with metadata here to get this much variation.</p>



<a name="207596300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596300">(Aug 21 2020 at 02:54)</a>:</h4>
<p><del>sometimes I wonder what we _do_ track on perf</del></p>



<a name="207596304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596304">(Aug 21 2020 at 02:54)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> Yeah, I feel like we ought to shove a version of <code>rustc_middle</code> into the perf testsuite. ;)</p>



<a name="207596310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596310">(Aug 21 2020 at 02:55)</a>:</h4>
<p>much easier said than done, the compiler loves to rely on unstable features and move with the tides</p>



<a name="207596319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596319">(Aug 21 2020 at 02:55)</a>:</h4>
<p>but I do miss irsy :(</p>



<a name="207596327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596327">(Aug 21 2020 at 02:55)</a>:</h4>
<p>a bygone era, I suppose. a simpler time :P</p>



<a name="207596332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596332">(Aug 21 2020 at 02:55)</a>:</h4>
<p>Well, in any case...</p>



<a name="207596334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596334">(Aug 21 2020 at 02:55)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> do you happen to know by any chance how long stage 1 takes before/after?</p>



<a name="207596374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596374">(Aug 21 2020 at 02:56)</a>:</h4>
<p>you might be able to see a win there :D</p>



<a name="207596377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596377">(Aug 21 2020 at 02:56)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> I don't have that handy, but it'd be easy for me to test it, if you can suggest the right approach for timing it.</p>



<a name="207596378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596378">(Aug 21 2020 at 02:56)</a>:</h4>
<p>sadly it probably requires a full build</p>



<a name="207596383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596383">(Aug 21 2020 at 02:56)</a>:</h4>
<p>/me build-tested on a 96-way, and would be happy to take another 10 minutes for two full builds. ;)</p>



<a name="207596386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596386">(Aug 21 2020 at 02:56)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> just uhh <code>rm -rf build/*/stage1*</code> and <code>./x.py build</code></p>



<a name="207596393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596393">(Aug 21 2020 at 02:57)</a>:</h4>
<p>it should print the relevant times as it goes</p>



<a name="207596404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596404">(Aug 21 2020 at 02:57)</a>:</h4>
<p>Also, I'm all for trying uncompressed metadata too. I feel like I saw that tried at some point.</p>



<a name="207596407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596407">(Aug 21 2020 at 02:57)</a>:</h4>
<p>That'd take a little more work, to actually mmap rather than reading into memory.</p>



<a name="207596454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596454">(Aug 21 2020 at 02:58)</a>:</h4>
<p>I think what you're decompressing is from a <code>mmap</code>'d <code>.so</code></p>



<a name="207596459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596459">(Aug 21 2020 at 02:58)</a>:</h4>
<p>Ah, interesting. So we'd just need to stop copying, and reference in-place?</p>



<a name="207596474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596474">(Aug 21 2020 at 02:59)</a>:</h4>
<p>I guess you might need to keep the ownership of the LLVM object accessing the dylib (assuming it's still that), around somehow, for longer, but that shouldn't be too hard</p>



<a name="207596479"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596479" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596479">(Aug 21 2020 at 02:59)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> It looks like doing that rm and then build just built std, not the compiler.</p>



<a name="207596525"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596525" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596525">(Aug 21 2020 at 03:00)</a>:</h4>
<p>hmpf</p>



<a name="207596530"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596530" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596530">(Aug 21 2020 at 03:00)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> add <code>--stage 2</code> I guess</p>



<a name="207596535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596535">(Aug 21 2020 at 03:00)</a>:</h4>
<p>or add <code>build src/rustc</code></p>



<a name="207596538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596538">(Aug 21 2020 at 03:01)</a>:</h4>
<p>I'd rather avoid that if possible</p>



<a name="207596545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596545">(Aug 21 2020 at 03:01)</a>:</h4>
<p>just so we know it roughly matches what we'd see e.g. on CI</p>



<a name="207596553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596553">(Aug 21 2020 at 03:01)</a>:</h4>
<p>CI does <code>build --stage 2 src/rustc</code></p>



<a name="207596554"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596554" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596554">(Aug 21 2020 at 03:01)</a>:</h4>
<p>that's why I originally added the flag</p>



<a name="207596562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596562">(Aug 21 2020 at 03:02)</a>:</h4>
<p>but that's building rustc thrice</p>



<a name="207596599"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596599" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596599">(Aug 21 2020 at 03:02)</a>:</h4>
<p>yes</p>



<a name="207596600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596600">(Aug 21 2020 at 03:02)</a>:</h4>
<p>that used to be the default lol</p>



<a name="207596606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596606">(Aug 21 2020 at 03:02)</a>:</h4>
<p>no? you have to opt into it in <code>config.toml</code> for that to be the case</p>



<a name="207596610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596610">(Aug 21 2020 at 03:02)</a>:</h4>
<p>oh I see</p>



<a name="207596616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596616">(Aug 21 2020 at 03:02)</a>:</h4>
<p>since almost nobody wants that heh</p>



<a name="207596619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596619">(Aug 21 2020 at 03:03)</a>:</h4>
<p>Waiting on rustc_middle...</p>



<a name="207596626"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596626" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596626">(Aug 21 2020 at 03:03)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> anyway, you want the first <code>Finished release ...</code> after:</p>
<div class="codehilite"><pre><span></span><code>Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -&gt; x86_64-unknown-linux-gnu)
</code></pre></div>



<a name="207596673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596673">(Aug 21 2020 at 03:04)</a>:</h4>
<p>make sure to run the <code>rm</code> command before builds if you change the source, I think there's some bug in <code>x.py</code></p>



<a name="207596676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596676">(Aug 21 2020 at 03:04)</a>:</h4>
<p>I did. I'll do so again after reverting the patch.</p>



<a name="207596738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596738">(Aug 21 2020 at 03:06)</a>:</h4>
<div class="codehilite"><pre><span></span><code>Finished release [optimized] target(s) in 3m 17s
</code></pre></div>



<a name="207596741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596741">(Aug 21 2020 at 03:06)</a>:</h4>
<p>That's with the patch. One moment, checking without...</p>



<a name="207596745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596745">(Aug 21 2020 at 03:07)</a>:</h4>
<p>ah that's quick enough that you can do several runs, nice</p>



<a name="207596852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596852">(Aug 21 2020 at 03:10)</a>:</h4>
<p>Because of the change I reverted, it's rebuilding part of stage0 before building stage 1.</p>



<a name="207596911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596911">(Aug 21 2020 at 03:12)</a>:</h4>
<p>that shouldn't impact the stage1 time</p>



<a name="207596934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596934">(Aug 21 2020 at 03:13)</a>:</h4>
<p>It doesn't, it just means I'm waiting longer. ;)</p>



<a name="207596937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596937">(Aug 21 2020 at 03:14)</a>:</h4>
<p>Worked though.</p>



<a name="207596962"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596962" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596962">(Aug 21 2020 at 03:14)</a>:</h4>
<div class="codehilite"><pre><span></span><code>Finished release [optimized] target(s) in 3m 15s
</code></pre></div>



<a name="207596978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596978">(Aug 21 2020 at 03:14)</a>:</h4>
<p>Looks like a wash.</p>



<a name="207596983"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596983" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596983">(Aug 21 2020 at 03:14)</a>:</h4>
<p>Can't spend much time waiting on compression if you're doing it on 96 CPUs in parallel. ;)</p>



<a name="207596993"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207596993" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207596993">(Aug 21 2020 at 03:15)</a>:</h4>
<p>was about to say, yeah. you might need to do it a few times and see if there's a pattern, I was hoping it'd be faster tbh</p>



<a name="207597057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597057">(Aug 21 2020 at 03:16)</a>:</h4>
<p>you could try <code>./x.py build; rm -rf build/*/stage1*; ./x.py build --stage 2 -j 8</code> to build slower but likely get more interesting numbers, too</p>



<a name="207597061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597061">(Aug 21 2020 at 03:16)</a>:</h4>
<p>First I'm getting a second set of numbers.</p>



<a name="207597068"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597068" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597068">(Aug 21 2020 at 03:17)</a>:</h4>
<p>Want to make sure that, for instance, the first build wasn't disadvantaged by being the first thing I did after booting up the system.</p>



<a name="207597201"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597201" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597201">(Aug 21 2020 at 03:21)</a>:</h4>
<p>Another build without the patch:     Finished release [optimized] target(s) in 3m 17s</p>



<a name="207597377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597377">(Aug 21 2020 at 03:27)</a>:</h4>
<p>And another with the patch:     Finished release [optimized] target(s) in 3m 15s</p>



<a name="207597379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597379">(Aug 21 2020 at 03:27)</a>:</h4>
<p>So yeah, definitely a wash on a 96-way rustc build, since the times switched. ;)</p>



<a name="207597492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207597492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207597492">(Aug 21 2020 at 03:31)</a>:</h4>
<p><span class="user-mention" data-user-id="119009">@eddyb</span> Thoughts?</p>



<a name="207598677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598677">(Aug 21 2020 at 04:01)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I'd look into why you get any non-<code>-doc</code> perf changes, maybe <code>perf</code> builds some <code>dylib</code>s</p>



<a name="207598722"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598722" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598722">(Aug 21 2020 at 04:02)</a>:</h4>
<p>/me is a little confused. Isn't this metadata we produce for every crate?</p>



<a name="207598791"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598791" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598791">(Aug 21 2020 at 04:04)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> we don't compress it normally, only when building a <code>dylib</code>. which is only used for rustc itself in practice</p>



<a name="207598802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598802">(Aug 21 2020 at 04:04)</a>:</h4>
<p>oh we switched that to only one crate, didn't we. so yeah my bad it shouldn't matter in practice</p>



<a name="207598811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598811">(Aug 21 2020 at 04:04)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> e.g. <code>ls ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/*.so</code></p>



<a name="207598879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598879">(Aug 21 2020 at 04:06)</a>:</h4>
<p>all of those (except <code>libLLVM</code> since that's not Rust) are runtime dependencies of <code>rustc</code> and only <code>librustc_driver-*.so</code> is really used as a Rust dependency in <code>dylib</code> form. so yeah</p>



<a name="207598889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598889">(Aug 21 2020 at 04:07)</a>:</h4>
<p>Interesting. How easy would it be to switch to compressing it for everything (again), and see if that gives a bigger performance win?</p>



<a name="207598910"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598910" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598910">(Aug 21 2020 at 04:07)</a>:</h4>
<p>I can almost guarantee it's a performance loss, since you're strictly doing more work. it'd be less space on disk, but I don't think anything we measure is proportional to that</p>



<a name="207598963"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598963" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598963">(Aug 21 2020 at 04:08)</a>:</h4>
<p>as for how hard it would be: shouldn't be much harder than decompression is in the dylib path</p>



<a name="207598971"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598971" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598971">(Aug 21 2020 at 04:08)</a>:</h4>
<p>presumably once you decompress it doesn't even matter where the compressed data came from</p>



<a name="207598996"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207598996" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207598996">(Aug 21 2020 at 04:09)</a>:</h4>
<p>the existing <code>rlib</code> codepath should be called something with "archive" in the name (since it's just an <code>.a</code> file)</p>



<a name="207599041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599041">(Aug 21 2020 at 04:10)</a>:</h4>
<p>Nevermind then, I won't poke at that if you're saying it's almost certainly not going to help.</p>



<a name="207599052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599052">(Aug 21 2020 at 04:11)</a>:</h4>
<p>As for the rest, though, it's giving a rather visible performance improvement for non-doc builds.</p>



<a name="207599055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599055">(Aug 21 2020 at 04:11)</a>:</h4>
<p>/me ponders.</p>



<a name="207599105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599105">(Aug 21 2020 at 04:12)</a>:</h4>
<p>I don't see how they would, maybe proc macros relying on <code>dylib</code> metadata?</p>



<a name="207599106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599106">(Aug 21 2020 at 04:12)</a>:</h4>
<p>in which case we probably just want to not compress at all</p>



<a name="207599115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599115">(Aug 21 2020 at 04:13)</a>:</h4>
<p>(I think the only reason we compress the <code>dylib</code> case is making the rustc we ship smaller)</p>



<a name="207599354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599354">(Aug 21 2020 at 04:19)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> btw my intuition regarding "doing less work up-front is better": we've consistently shaved off "startup time" by making metadata reading more on-demand/random-access, and I don't think we can ever go back to anything proportional to the number of bytes in the metadata. if we ever compress again, it will have to at a much smaller scale, where we know we don't decompress most of the file most of the time</p>



<a name="207599423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599423">(Aug 21 2020 at 04:20)</a>:</h4>
<p>the more something is used, the more it helps to match what it needs to be used as, etc.</p>



<a name="207599454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599454">(Aug 21 2020 at 04:21)</a>:</h4>
<p>which reminds me, we use LEB128 still I think, and it might help to not do that in several places, depending on the kind of integer encoded. or we could have <code>u16</code> units instead of <code>u8</code> units</p>



<a name="207599468"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599468" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599468">(Aug 21 2020 at 04:21)</a>:</h4>
<p>classic time-space tradeoff though :P</p>



<a name="207599552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599552">(Aug 21 2020 at 04:24)</a>:</h4>
<p>Should I be getting anything of value out of <a href="https://perf.rust-lang.org/detailed-query.html?commit=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;base_commit=e15510ca33ea15c893b78710101c962b11459963&amp;benchmark=webrender-wrench-check&amp;run_name=incr-unchanged">https://perf.rust-lang.org/detailed-query.html?commit=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;base_commit=e15510ca33ea15c893b78710101c962b11459963&amp;benchmark=webrender-wrench-check&amp;run_name=incr-unchanged</a> , or is it all noise?</p>



<a name="207599614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599614">(Aug 21 2020 at 04:25)</a>:</h4>
<p>all of the positive and negative differences make me doubt this is useful. although, <code>expand_crate</code> could be relevant? but isn't this for a run that got <em>faster</em>?</p>



<a name="207599629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599629">(Aug 21 2020 at 04:25)</a>:</h4>
<p>It is, yeah. The queries don't seem to reflect that, though.</p>



<a name="207599669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599669">(Aug 21 2020 at 04:26)</a>:</h4>
<p>Is the performance here all <em>outside</em> of queries somehow?</p>



<a name="207599674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599674">(Aug 21 2020 at 04:26)</a>:</h4>
<p>Here's a performance delta for a "full" run: <a href="https://perf.rust-lang.org/detailed-query.html?commit=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;base_commit=e15510ca33ea15c893b78710101c962b11459963&amp;benchmark=tokio-webpush-simple-check&amp;run_name=full">https://perf.rust-lang.org/detailed-query.html?commit=50541435b8c0b771394cb492b0b2fbdbc3c3ddd9&amp;base_commit=e15510ca33ea15c893b78710101c962b11459963&amp;benchmark=tokio-webpush-simple-check&amp;run_name=full</a></p>



<a name="207599675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599675">(Aug 21 2020 at 04:26)</a>:</h4>
<p>I think this is just time being near-useless</p>



<a name="207599678"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599678" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599678">(Aug 21 2020 at 04:26)</a>:</h4>
<p>hence all my obsession with <code>rdpmc</code></p>



<a name="207599694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599694">(Aug 21 2020 at 04:27)</a>:</h4>
<p>Fair.</p>



<a name="207599744"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599744" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599744">(Aug 21 2020 at 04:28)</a>:</h4>
<p>Any hypotheses on why this might be showing as faster across the board (other than -doc)? That doesn't seem like it'd be noise.</p>



<a name="207599758"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599758" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599758">(Aug 21 2020 at 04:29)</a>:</h4>
<p>my best guess is proc macros (which are a hybrid of <code>cdylib</code> and <code>dylib</code>)</p>



<a name="207599764"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599764" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599764">(Aug 21 2020 at 04:29)</a>:</h4>
<p>wait I can go check</p>



<a name="207599796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599796">(Aug 21 2020 at 04:30)</a>:</h4>
<p>Oh?</p>



<a name="207599831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599831">(Aug 21 2020 at 04:31)</a>:</h4>
<p><code>readelf -x .rustc ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libchalk_derive-*.so | less</code></p>



<a name="207599841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599841">(Aug 21 2020 at 04:31)</a>:</h4>
<p>yeah it's compressed</p>



<a name="207599844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599844">(Aug 21 2020 at 04:31)</a>:</h4>
<p>so my suggestion would be disabling compression for at least proc macros</p>



<a name="207599945"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207599945" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207599945">(Aug 21 2020 at 04:34)</a>:</h4>
<p>So, you think it's providing a win to switch to faster compression, but you think it'd provide an even better win to stop compressing?</p>



<a name="207600134"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207600134" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207600134">(Aug 21 2020 at 04:41)</a>:</h4>
<p>right, I think proc macros are collateral damage from something we only really want to do to <code>librustc_driver-*.so</code></p>



<a name="207656084"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207656084" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207656084">(Aug 21 2020 at 16:41)</a>:</h4>
<p>Q: do any other filesystems other than zfs compress data, or have ability to, by default?</p>



<a name="207656656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207656656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207656656">(Aug 21 2020 at 16:46)</a>:</h4>
<p>According to <a href="https://en.wikipedia.org/wiki/Comparison_of_file_systems">wikipedia</a>: BeeGFS, APFS, NTFS, F2FS, Reiser4, NSS, NWFS, Fossil, ZFS, Btrfs, SquashFS</p>



<a name="207671964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207671964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jubilee <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207671964">(Aug 21 2020 at 19:05)</a>:</h4>
<p>Btrfs and Reiser4 definitely, they attempt a similar conceptual model as ZFS though I believe they differ on several points.</p>



<a name="207674543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207674543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207674543">(Aug 21 2020 at 19:28)</a>:</h4>
<p>Also worth noting: there are two main reasons for a filesystem to support compression. Some filesystems support compression to save space; they might sacrifice performance to do so. Some filesystems support compression to improve performance, using a compression format like lz4 or similar that's pretty much doable at the speed of memory.</p>



<a name="207675249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207675249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207675249">(Aug 21 2020 at 19:34)</a>:</h4>
<p>Sort of related, Fedora is enabling swap-on-zram soon:<br>
<a href="https://fedoraproject.org/wiki/Changes/SwapOnZRAM">https://fedoraproject.org/wiki/Changes/SwapOnZRAM</a></p>



<a name="207675562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207675562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207675562">(Aug 21 2020 at 19:37)</a>:</h4>
<p>(and <a href="https://github.com/systemd/zram-generator">systemd/zram-generator</a> is written in Rust!)</p>



<a name="207706243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/using%20snappy%20for%20metadata/near/207706243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jubilee <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/using.20snappy.20for.20metadata.html#207706243">(Aug 22 2020 at 04:05)</a>:</h4>
<p>Yeah, ZFS as I understand it is about filesystem integrity above all else, compression simply is something ZFS can slot in automagically as part of how it works, along with deduplication and etc.</p>
<p>"We basically built this to have a spot for middleware because we wanted to be able to apply our own middleware and then it was mostly adding a few extra logic slots for middleware."</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>